Skip to content

Conversation

tracelarue
Copy link

Gemini Live for low-latency bidirectional voice interactions with ros-mcp-server.

  • Added gemini_live to examples/2_gemini.
  • Enables audio input from the user and audio output from Gemini.
  • Enables Gemini Live to use ros-mcp-server.
  • Tested in ubuntu 22.04, python 3.10, ros2 humble.

@tracelarue tracelarue changed the title Voice interactions with Gemini Live and ros-mcp-server added to Gemini example. Add voice interactions with Gemini Live and ros-mcp-server to Gemini example. Sep 24, 2025
@stex2005 stex2005 requested review from lpigeon, rjohn-v and stex2005 and removed request for lpigeon September 24, 2025 16:05
@stex2005
Copy link
Collaborator

Thank you for your contribution, @tracelarue. I will give it a try soon.

@rjohn-v — I’d suggest adding a client/ folder in the repository to store installation packages and runnable clients. This would help keep different client implementations (e.g., Gemini API client) and their installation steps organized in one place. I’m not sure I’d keep this under examples/.

@stex2005
Copy link
Collaborator

I connected issue robotmcp/robot-mcp-client#20 to this PR.
After, we can close it and reopen for other APIs.

@stex2005
Copy link
Collaborator

stex2005 commented Sep 26, 2025

image

Raises this error during installation, seems that I need a system-package: portaudio.

I woudl recommend trying to include this into dependencies in the README.md + comamnd to install:

sudo apt install portaudio19-dev

@stex2005
Copy link
Collaborator

image

Couldn't run uv run on my WSL Ubuntu. Please specify that this works only on Ubuntu, will try on Ubuntu soon.

@stex2005
Copy link
Collaborator

@tracelarue @rjohn-v Another good next step would be to provide a dockerized version of the Gemini client, so it can be run more easily in different environments. Since the client is only a tool within this project, I don’t think we should invest too much effort in tightly integrating it into the repo. A simplified version of client_gemini (without audio support) would already be a good, lightweight solution and would find a good place in clients folder.

@stex2005
Copy link
Collaborator

Couldn't run uv run on my WSL Ubuntu. Please specify that this works only on Ubuntu, will try on Ubuntu soon.

image

This is the same error when I try to run mcp_handler.py

@tracelarue
Copy link
Author

@stex2005 Thank you for the review and feedback. I'll work on getting these changes and fixes implemented.

@mokcontoro
Copy link
Contributor

@tracelarue wow, voice command sounds super cool. thanks for your contributions. I cannot wait for trying this feature soon!

@tracelarue tracelarue requested a review from stex2005 October 4, 2025 19:14
@tracelarue
Copy link
Author

@stex2005 Ready for review

Changes made:

  • removed mcp_config.json and instruct user to create their own
  • removed pyproject.toml and uv.lock. Now uses uv pip install to install dependencies in the existing ros-mcp .venv
  • updated README.md to specify .env location
  • renamed gemini_live.py to gemini_client.py
  • specified it only works on ubuntu in README.md (try WSL again after these changes)
  • removed mcp_handler.py in favor of Google's latest method to connect to mcp servers
  • Other gemini_client.py improvements and README.md updates

@stex2005
Copy link
Collaborator

stex2005 commented Oct 5, 2025

@stex2005 Ready for review

Changes made:

  • removed mcp_config.json and instruct user to create their own
  • removed pyproject.toml and uv.lock. Now uses uv pip install to install dependencies in the existing ros-mcp .venv
  • updated README.md to specify .env location
  • renamed gemini_live.py to gemini_client.py
  • specified it only works on ubuntu in README.md (try WSL again after these changes)
  • removed mcp_handler.py in favor of Google's latest method to connect to mcp servers
  • Other gemini_client.py improvements and README.md updates

@tracelarue thank you for the contribution. I will try it :)

@stex2005
Copy link
Collaborator

stex2005 commented Oct 5, 2025

@tracelarue thanks again for your contribution.

I've tried your example on WSL and Ubuntu and it still gives some compatibility issues due to hardware/driver changes on camera/microphone. I would suggest making this example a simplified version of the gemini_client that works only with text input/output. At this moment the example code is too broad and complex to be maintained.

I am working on this simplified example and I will soon share it.

@stex2005
Copy link
Collaborator

stex2005 commented Oct 6, 2025

@tracelarue thanks again for your contribution.

I've tried your example on WSL and Ubuntu and it still gives some compatibility issues due to hardware/driver changes on camera/microphone. I would suggest making this example a simplified version of the gemini_client that works only with text input/output. At this moment the example code is too broad and complex to be maintained.

I am working on this simplified example and I will soon share it.

@tracelarue Thank you so much again for this contribution. We are debating whether client implementations should be in this or another repository. I would keep this PR on hold until we decide.

Meantime, I would suggest working on the gemini_client lite version, to have it work on both Ubuntu machines and with WSL.

@tracelarue
Copy link
Author

@stex2005 Thanks for the review. I’ll start working on a text-only version and test it on WSL for compatibility. Could you share your hardware setup and specify where the issue occurred? Was the gemini_client able to launch and connect to the MCP? I’ve previously run the live version on a Raspberry Pi, where I had to adjust the mic and speaker sampling rates for hardware compatibility.

@stex2005
Copy link
Collaborator

stex2005 commented Oct 7, 2025

@stex2005 Thanks for the review. I’ll start working on a text-only version and test it on WSL for compatibility. Could you share your hardware setup and specify where the issue occurred? Was the gemini_client able to launch and connect to the MCP? I’ve previously run the live version on a Raspberry Pi, where I had to adjust the mic and speaker sampling rates for hardware compatibility.

This is the first attempt: https://github.com/stex2005/ros-mcp-client

@stex2005 stex2005 closed this Oct 8, 2025
@stex2005
Copy link
Collaborator

stex2005 commented Oct 8, 2025

We are temporarily closing this PR. Will be implemented in ros-mcp-client

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add example for running a local/on-prem LLM with MCP

3 participants